Mott Physics on Helical Edges of 2D Topological Insulators 
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We study roles of electron correlations on topological insulators on the honeycomb lattice with the 
spin-orbit interaction. Accurate variational Monte Carlo calculations show that the increasing on- 
site Coulomb interactions cause a strong suppression of the charge Drude weight in the helical-edge 
metallic states leading to a presumable Mott transition from a conventional topological insulator to 
an edge Mott insulator before a transition to a bulk antiferromagnetic insulator. The intermediate 
bulk-topological and edge-Mott-insulator phase has a helical spin-liquid character with the protected 
time-reversal symmetry. 



-Introduction. Recently, spin Hall insulators and its 
generalization, topological insulators (TIs) have at- 
tracted much attention as a new state of matter A re- 
markable feature of the newly discovered quantum phase 
is the ^2-type topological distinction from other conven- 
tional phases as well as the existence of robust gapless 
edges or surface states concomitant with the bulk insu- 
lating gap, which are all protected by the time reversal 
(TR) symmetry. The edge or surface modes of TI pro- 
vide us with truly one- or two-dimensional gapless and 
metallic electronic states. 

It has also been proposed that TI may appear in sys- 
tems under substantial electron correlations such as in Ad 
or 5d transition metal oxides [H-Q, while the interplay of 
electron correlations with the topological insulator has 
not been well understood, although the absence of the 
back scattering protected by the time reversal symmetry 
is expected to suppress electron correlation effects [l|, 0,9- 

In this Letter, based on results of calculations obtained 
from a multi- variable variational Monte Carlo (MVMC) 
methods improved by Tahara and one of the authors Q, 
we propose that electron correlation effects introduced by 
an onsite interaction, namely, a Hubbard U in the Kane- 
Mele model on the honeycomb lattice allow a transition 
from the above TI to an unconventional TI phase charac- 
terized by the charge gapful (insulating) but spin gapless 
edge excitations with a nonzero spin Drude weight within 
the same preserved topological nontriviality of the bulk 
states that are protected by the time reversal symmetry. 
This new topological edge Mott insulator (TEMI) phase 
is stabilized in a region of the intermediate correlation 
strength sandwiched by a bulk antiferromagnetic insula- 
tor (BAFI) with the broken time reversal symmetry (or 
bulk Mott insulator (BMI)) in the larger U region and 
the simple TI insulator in the weak correlation region. 

-Model. Wc study a tight binding hamiltonian on the 
two-dimensional honeycomb lattice proposed by Kane 
and Mele [l[ with inclusion of the on-site Coulomb inter- 
action, and without the Rashba term to study electron 
correlation effects on the topological insulator. Hereafter 
we call this simple model the Hubbard-Kane-Mele model 



and is defined as 
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where T-Lkm is the Kane-Mele hamiltonian and U gen- 
erates an onsite Hubbard interaction between the up 
and down spin electrons. Here we define i>ij = di x 

dj/ di x dj , and I = (i,a) (a = A, B) (see FigQ]). 

-Method. We perform unrestricted Hartree-Fock 
(UHF) calculations as well as MVMC calculations by op- 
timizing a large number of variational parameters. 




FIG. 1: Honeycomb lattice on which HKM model is defined. 



In the UHF calculation, we decouple the U term as 
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For the MVMC calculations, we employ a variational 
wave function Q defined as 

\^P) = PcWpair), (4) 

where Vq is the Gutzwiller factor defined by 

Vg = exp[- ^ 9initfi Iir } , (5) 
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and V] is the Jastrow factor defined by 



Vi = exp 
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with the spatially dependent variational parameters gi 
and vij. We impose the Gutzwiller factor on all the sites, 
whereas introduce the Jastrow factor only along the zig- 
zag edges. The one-body part \4>p&ti) is a generalized 
pairing wave function defined as 
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with fij being the complex variational parameters. In 
this study, we allow to have 2-sublattice (2 x L y - 
sublattice) structure or equivalently we have 2 X 2 X N s 
(2xL y x2x ) variational parameters for the torus 

(cylinder) . All the variational parameters are simultane- 
ously optimized by using the stochastic reconfiguration 
method [ltj generalized for complex variables. The 
accuracy of this method has been proven in a number of 
benchmarks 0, [ijj 

Charge and spin Drudc weights are calculated by in- 
troducing vector potentials as the Peierls factors, 



tija tua exp[iA a ■ f/j] 



(8) 



where fj = nid x + mid y and r/j = fj — fj. Here ni 
and mj are integers, and lattice vectors are a x and a y 
(see FigH]). From this Peierls factor, the charge and spin 
Drude weights are calculated from the energy stiffness 
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To clarify the edge state, we employ a cylinder with sizes 
N s =L x xL y x2, for the honcycome lattice with two sites 
on a unit cell and the periodic (free) boundary condi- 
tions in the x (y) directions. We have confirmed that the 
employed width L y is large enough to make isolated two 
edges at the two free boundaries at y = and y = L y . 
For the bulk properties we employ the torus, where the 
boundary is periodic for all the directions . 

-Bulk phase diagram. The ground state phase dia- 
gram of the bulk is shown in FigJ3] for the HKM model 
on the torus. Our MVMC results show the antiferromag- 
netic transition at U = U^ F ~ It for t 2 = O.lt. Below 
U ~ 7t, the bulk stays as a topological insulator and the 
peak height of the magnetic structure factor defined by 



SAF(q) = E ^XlXjl ■ Sj, 

61 ^ s I, J 



(11) 
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FIG. 2: (color online). Phase diagram of HKM model ob- 
tained by MVMC containing three phases for ts = O.lt. The 
bulk on the torus shows a phase transition from TI to the 
bulk antiferromagnetic insulator (BAFI) at U/t ~ 7. On the 
cylinder, the edge shows insulating behaviors for 3i < U. The 
intermediate TEMI is a gapless spin liquid at the edges. 



for the spin-1/2 operator Si scales to a size-independent 
constant after the size extrapolation in contrast to the 
Bragg peak height proportional to N s = L x x L y x 2 
observed in BAFI, as is shown in Fig|3] Here we have 
shown the peak values, which appear at the wavenumber 
(7 = and for the staggered modes within the unit cell, 
namely A/ = +1(-1) for I = (i,A) (I = (i,B)). The 
MVMC calculation gives the critical value of U for t-2 ^ 
larger than that for t<i = 0. These results are consistent 
with an auxiliary-field quantum Monte Carlo simulation 
for t 2 =0, which shows C/ c AF =4.3t[l2j. 

Here the magnetic moments in the BAFI phase align 
in the xy-plane. This fact conforms with the UHF re- 



sults (see also Ref|l3|) and the effective hamiltonian at 
the strong coupling limit; the second order perturbation 
of the second-neighbor hopping iti yields the second- 
neighbor exchange coupling as J 2 [SfSj — SfSJ — SfSj], 
where J 2 = 4t%/U. In the xy-plane, J 2 gives the fer- 
romagnetic coupling and stabilizes the BAFI moment 
within this plane. The non-zero magnetic moment within 
the xy-plane always opens a gap at the edge modes, al- 
though an infinitesimal magnetic moment along the z- 
axis does not open a gap. 

-Edge transport. We show MVMC results for the 
Drudc weights for the HKM model on the cylindrical ge- 
ometry with two zig-zag edges along the x-direction. A 
nonzero Drude weight D c defined in EQ) represents the co- 
herent charge transport or metalicity[lj|. If we introduce 
spin-dependent vector potentials, A a = o~A, we obtain 
the Drude weight for the spin channel namely the 
spin Drude weight. A direct evaluation of the Z 2 topolog- 
ical number proposed by Lee and Rvu[l6j could be done 
by using the same procedure. However, it requires much 
more computational cost, and is left for future studies. 

In FiglU we compare the results for charge and spin 
Drude weights by MVMC with those of the UHF ap- 
proximation. The data for 15 x 5 x 2 well represent 
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FIG. 3: (color online), (a) [/-dependence of Saf(? = 0) for 
various system sizes, (b) 1/N„ dependences of S , af(<? = 0). 
The inset shows the same quantity vs. l/-y/ N s /2 (= l/L x ). 
We note that Saf /JV s may be scaled linearly with 1/N S in the 
asymptotic region (l/N s — > 0) of the disordered phase, as we 
see in the main panel of Fig. 3(b), whereas as is known in the 
spin wave theory, it should be scaled linearly with 1/L X with 
a nonzero offset in the asymptotic region of the ordered phase, 
as we see in the inset of Fig. 3(b). These scalings suggest that 
the phase transition occurs between U = 6.75t and U — It. 
Error bars are all within the symbol sizes. 



the thermodynamic limit of the nonmagnetic UHF so- 
lution. We see consistent suppression (enhancement) of 
the charge (spin) Drude weight D C (D S ) arising from the 
increasing on-site Coulomb interaction U. Moreover, the 
suppression (enhancement) of D C (D S ) has nearly linear 
dependences on U ft. The renormalized dependences of 
D c (U r /t)/D c {0) vs. U r /t with U r = U for MVMC and 
U r ~ 1.4 x [/ for UHF make all of them universal, in- 
dicating that the thermodynamic limit for the MVMC 
is extracted from the UHF at the same U r as is shown 
in the lower panel of Figf4] The MVMC results support 
a transition on the edge from the TI to a charge gap- 
ful (insulating) phase (TEMI) around Uf M ~ 3i in the 
thermodynamic limit. 

The suppression and enhancement in the Drude 
weights are naturally accounted by focusing on the spin 
and charge pumping caused by the vector potentials. 
The spin-independent (spin-dependent) vector poten- 
tial causes the spin (charge) pumping along the zig-zag 
edges [H, which is nothing but the celebrated quantum 
spin Hall effects in the Kanc-Mclc model. Without the 



Rashba term, which mixes the spin-up and -down com- 
ponents, the spin-independent vector potential causes a 
spin pumping of the ^-components. Here we note that the 
small amount of spin accumulation of the ^-component 
does not induce a gap opening at the edge modes. 

The spin pumping generates spin polarization along 
the z-axis at the edges, which helps electrons to reduce 
the cost of the Hubbard U. Contrarily, the charge pump- 
ing forces to increase the double occupation at the edges 
resulting in the cost of U. Therefore, the energy increase 
with increasing vector potential (namely the stiffness or 
the Drude weight as the quadratic coefficients) decreases 
(increases) compared with E of the non-interacting sys- 
tem. Such spin-charge separated Drude weights appear 
even in the restricted Hartree-Fock calculation that does 
not allow the magnetic moments within the sy-plane. 
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FIG. 4: (color online). Upper left panel: U dependence 
of renormalized Drude weights. Upper right panel: Vector 
potential dependences of total energy E for L x = 5, L y — 3. 
Lower panel: Renormalized D c (U r /t)/D c (0) of the UHF and 
MVMC. The solid curves for the UHF data scales identically 
with the MVMC data for the same sizes when we employ the 
renormalized interaction U r . 

-Edge phase diagram. Based on the calculated spin 
and charge Drude weights, we show the edge phase di- 
agram in FigO It supports a metal-insulator transition 
at the edge at U = C/ ( f M ~ 3t, where the bulk still con- 
tinues to be paramagnetic TI beyond it. In contrast, the 
coherent edge spin transport is enhanced by increasing 
U. 

The low-energy effective model, namely, the helical 
Tomonaga-Luttinger liquid (HTLL) does not include the 
back scattering and Umklapp channels @,H[; which are, in 
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general, essential for the formation of insulating phases. 
However, for t 2 = O.li and U ~ 3t, the amplitude of 
the bulk gap, which limits the energy scale to justify 
the treatment by the topological band insulator, becomes 
comparable with U. Then, HTLL will fail to capture this 
Mott insulating behaviors of the edge modes. In fact, a 
large Coulomb repulsion (U ^> t, t 2 ) inevitably prohibits 
coherent propagations of electrons. In reality, it actu- 
ally results in gapful charge excitations for t 2 = O.li and 
f/ c EM (~ M) <U < f/ c AF (~ 7t) with gapless spin excita- 
tions at the edge for the HKM. 

- Whole phase diagram As we see in FigH through the 
U variation, the bulk state is always insulating (charge 
gapful) if the spin-orbit interaction is nonzero while it has 
a spin gapless excitation exclusively in the conventional 
BMI or BAFI phase at the largest U region. However, 
the edge state is always characterized by the gapless spin 
excitations while the charge excitation is gapless only in 
the lowest U region of the TI phase. Then we find the 
intermediate phase, TEMI where the bulk excitations 
are gapful in both spin and charge channels, while in 
the edge state the charge excitation is gapful (insulat- 
ing) and the spin excitation is gapless with a spin liq- 
uid behavior. We note this general phase diagram with 
three phases contained may be universal also in three- 
dimensional systems except for the additional possibility 
that, depending on the lattice geometry, the spin liquid 
in the TEMI phase could be replaced with the magnetic 
symmetry breaking such as the antiferromagnetic order 
at the edge (surface) as is proposed by ref.[6(. 

-Discussion A possible interpretation of the TEMI 
phase is a spinon liquid under the fractionalizcd electrons 
in the slave-rotor approximation proposed by Young et 
and Pesin and BalentsQ. However, the ID spin- 
charge separation in the simple Hubbard chain is charac- 
terized by the spin/charge excitations as bosonic collec- 
tive modes of the Tomonaga-Luttinger liquid 17. flj|. In 
the present helical spin liquid on the edges, it is also likely 
to have gapful charge and gapless spin bosonic collective 
density modes leading to the HTLL B, B distinct from 
the chiral Tomonaga-Luttinger liquid [18j in the quan- 
tum Hall phase. 

The conventional TI in the cylinder geometry is char- 
acterized by the nonzero diagonal charge (spin) conduc- 
tivities denoted by cr ccxx =/= (u SSX x ^ 0) and nonzero 
spin-charge transverse conductivity denoted by a csxy = 



Vscxy 7^ and a c 



o~scyx 7^ 0, where all arc solely from 



the edge contributions. Other spin-charge off-diagonal el- 
ements are zero. On the other hand, the present TEMI 
with the gapful charge and gapless spin liquid edges keeps 
o~ S sxx 7^ again contributed only from the edge, whereas 
all the other elements including a CCXXl a, 



csxy 



' sexy 



1 csyx 



and 

o~ S cyx vanish. The Onsager reciprocal relation of 
course always holds. In the both TI and TEMI phases, 
all the bulk conductivities vanish, while in the BMI (or 
BAFI) phase, the bulk and edge spin conductivities may 



remain nonzero with all the other linear responses involv- 
ing the charge part vanish irrespective of bulk or edge. 

-Summary Our present variational Monte Carlo cal- 
culations show that the local electron correlation U of 
the Hubbard-Kane-Mele model on the honeycomb lattice 
drives a strong crossover or a quantum phase transition 
within the topologically nontrivial phase. The transition 
appears to separate an edge metallic TI phase at lower 
U from TEMI phase with charge gapful and spin gap- 
less (spin liquid) edges at larger U, where a bulk charge- 
spin gap is always retained through these two topological 
phases. Namely, the larger U phase is characterized by a 
vanishing charge Drude weight together with a nonzero 
and large spin Drude weight in contrast to the both large 
charge and spin Drude weights in the lower U phase. 
With further increase of U, this TEMI phase undergoes 
a transition into the BAFI phase with the time reversal 
symmetry breaking. 
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